A Survey of New York City’s Airbnb Listings

R for beginners: a data analysis workshop

Goals

• Learn to think like a data analyst
• Learn the basics of R programming language
• Learn some powerful techniques for analyzing and exploring a dataset

we’ll do these by taking a close look at some interesting data: Airbnb listings for New York City in May 2017.

It’s okay if you can’t start using these techniques right away after the workshop. To be comfortable with a tool like R takes some time and practice. But we do hope that your exposure today to these tools and techniques will open your eyes about possibilities and motivate you to keep learning.

RStudio

RStudio is a free program that makes writing R code much more enjoyable and efficient.

It has four main panes. This one is the code editor (also known as script editor and source editor). This is where we’ll spend most of our time. We’ll say more about the other panes when the time comes.

R Notebook

Before starting the analysis, let’s understand how this document works. I’m assuming you’re reading this on RStudio.

This is an R Notebook. An R Notebook contains commentary interspersed with code chunks. The code chunks can be run (executed) independently and interactively. The output will appear below the code chunk. R Notebooks are easy to convert to well-formatted final documents as a pdf file, a webpage, or an MS Word file. More here

Click on the preview button above to get an idea.

What you’re reading is the commentary and what you see below is the container for a code chunk.

# The code goes here

To run a code chunk, click on the little triangle at the right edge of the code chunk. Or, use the keyboard shortcut CTRL + SHIFT + ENTER (Windows) or CMD + SHIFT + ENTER (Mac).

Exercise: Run the chunk below and see what happens.

a <- 5 + 7
print(a)

[1] 12

As you see, the output appears under the code. This way you can immediately see the result of the code you write at all steps of the analysis.

(Do you know what’s going on in the code above? We’ll talk about variables and assignments later.)

Feel free to add your commentary and code anywhere in this document. You can always download the unmodified version from here.

To write your own code chunk, look for the insert button above and then select R. Or, use the keyboard shortcut CTRL + ALT + I (Windows) or CMD + OPTION + I (Mac).

Exercise: Place the cursor below and create a container for writing code. Now write some code and run it. For example, find the result of 3543 / 562.

The data

The dataset contains almost all the listings in NYC in May 2017.The data came from Inside Airbnb.

It’s always a good idea to approach a new dataset with a few basic questions. Some examples:

• What kind of data is here? What are the variables? How many observations?
• Who collected these data? How? Why?
• How accurate are these data?
• How complete are these data? Are there too many missing values?
• Are there anomalies that I need to be aware of?
• Do I have the legal rights to use these data? Under what conditions?

Here’s some background information that may answer some of these questions.

Let’s take a look at the data. Open the csv file in Excel. Browse around a bit. Any observations or questions?

Discuss: Can you explain what each column is about?

Discuss: Are there any data that you’d like to have but not there?

Discuss: What’s the first thing you’d like to find out from these data?

Discuss: What do you think about the quality of the data? Why?

The context

Data analysis happens within a larger context. Usually there’s an overarching business, policy, or scientific question that one would like to answer. Often that question is not very clear. Whatever the case, you need to approach the data with curiosity about the larger context.

The more you understand the context of the data, the better will be your questions and hypotheses guiding your analysis.

For our dataset, we should have a good understanding of the business model of Airbnb as well as the economy and geography of NYC.

Discuss: Is there anything else we should know about?

Airbnb

Discuss: How much do we need to know about Airbnb and its business? Is my personal experience with Airbnb enough? What if I don’t have any personal experience?

How Airbnb works

Example airbnb listing

New York City

We can start with a map of the city.

img

The map shows the relative size and location of the five boroughs of NYC.

Discuss: What else do we know about these boroughs? What about the neighborhoods within the boroughs?

Analysis

Install and load libraries

We’ll first install a few libraries that we’ll need at different stages of the analysis. This may take a while.

install.packages("lubridate")

Installing package into 㤼㸱C:/Users/mehedia/Documents/R/win-library/3.4㤼㸲
(as 㤼㸱lib㤼㸲 is unspecified)
trying URL 'https://mran.microsoft.com/snapshot/2017-05-01/bin/windows/contrib/3.4/lubridate_1.6.0.zip'
Content type 'application/zip' length 667329 bytes (651 KB)
downloaded 651 KB

package ‘lubridate’ successfully unpacked and MD5 sums checked

The downloaded binary packages are in
    C:\Users\mehedia\AppData\Local\Temp\Rtmpk3mp83\downloaded_packages

And then we load the libraries for our current R session. After these libraries are successfully loaded, all the functions in these libraries will be available for our use.

library(readr)
library(dplyr)

Attaching package: 㤼㸱dplyr㤼㸲

The following objects are masked from 㤼㸱package:stats㤼㸲:

    filter, lag

The following objects are masked from 㤼㸱package:base㤼㸲:

    intersect, setdiff, setequal, union

library(ggplot2)
library(stringr)
library(lubridate)

Attaching package: 㤼㸱lubridate㤼㸲

The following object is masked from 㤼㸱package:base㤼㸲:

    date

Load and prepare data

We’ll use the function read_csv, which comes from the library readr, to load data from file and convert the data into a dataframe. A dataframe is a tabular data structure, with columns as variables and rows as observations. In this case, exactly as it is in the csv file.

Discuss: What’a csv file? What other formats are out there for storing data? How can we load data from an unfamiliar format?

df <- read_csv("airbnb_newyork.csv")

Parsed with column specification:
cols(
  .default = col_integer(),
  host_since = col_character(),
  host_response_time = col_character(),
  host_response_rate = col_character(),
  neighbourhood = col_character(),
  borough = col_character(),
  property_type = col_character(),
  room_type = col_character(),
  bathrooms = col_double(),
  amenities = col_character(),
  price = col_number(),
  calendar_updated = col_character(),
  cancellation_policy = col_character(),
  listing_url = col_character(),
  description = col_character()
)
See spec(...) for full column specifications.

First thing to check if the data have been successully loaded and assigned to the variable (here df). We don’t see any error — which is a good sign. We can also see the variable df in the environment pane (usually to the right of this code editor pane – click on “Environment” if hidden).

Next thing to check is whether read_csv was able to correctly infer the data type of each variable.

The function glimpse will give us a better glimpse.

glimpse(df)

Observations: 40,752
Variables: 28
$ host_id                     <int> 58306608, 124280354, 124280354, 124280354, 124280354, 1242...
$ host_since                  <chr> "2/11/2016", "4/4/2017", "4/4/2017", "4/4/2017", "4/4/2017...
$ host_response_time          <chr> "within an hour", "within an hour", "within an hour", "wit...
$ host_response_rate          <chr> "0.78", "0.99", "0.99", "0.99", "0.99", "0.99", "0.95", "N...
$ host_listings_count         <int> 1, 7, 7, 7, 7, 7, 1, 1, 1, 7, 1, 1, 1, 1, 1, 1, 2, 2, 1, 1...
$ neighbourhood               <chr> "Hell's Kitchen", "Hell's Kitchen", "Hell's Kitchen", "Hel...
$ borough                     <chr> "Manhattan", "Manhattan", "Manhattan", "Manhattan", "Manha...
$ zip_code                    <int> 10000, 10001, 10001, 10001, 10001, 10001, 10001, 10001, 10...
$ property_type               <chr> "Apartment", "Apartment", "Apartment", "Apartment", "Apart...
$ room_type                   <chr> "Private room", "Shared room", "Shared room", "Shared room...
$ accommodates                <int> 2, 1, 1, 1, 1, 1, 2, 1, 1, 1, 2, 2, 1, 2, 1, 1, 2, 2, 1, 1...
$ bathrooms                   <dbl> 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 0.5, 3.0, 1.0, 1.0, 1.0, 1.0...
$ bedrooms                    <int> 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1...
$ beds                        <int> 1, 1, 1, 1, 1, 1, 2, 6, 1, 1, 1, 1, 1, 1, 1, 1, 2, 1, 1, 1...
$ amenities                   <chr> "{TV,Internet,\"Wireless Internet\",\"Air conditioning\",P...
$ price                       <dbl> 200, 28, 39, 39, 39, 43, 50, 50, 50, 53, 54, 55, 55, 60, 6...
$ calendar_updated            <chr> "never", "5 days ago", "5 days ago", "3 days ago", "6 days...
$ number_of_reviews           <int> 0, 0, 2, 1, 0, 3, 185, 0, 0, 0, 21, 5, 3, 12, 0, 20, 1, 1,...
$ review_scores_rating        <int> NA, NA, 100, 100, NA, 87, 93, NA, NA, NA, 96, 100, 80, 92,...
$ review_scores_accuracy      <int> NA, NA, 10, 10, NA, 9, 9, NA, NA, NA, 10, 10, 10, 9, NA, 1...
$ review_scores_cleanliness   <int> NA, NA, 10, 10, NA, 8, 9, NA, NA, NA, 10, 8, 8, 9, NA, 9, ...
$ review_scores_checkin       <int> NA, NA, 10, 10, NA, 9, 10, NA, NA, NA, 10, 10, 10, 9, NA, ...
$ review_scores_communication <int> NA, NA, 10, 10, NA, 9, 10, NA, NA, NA, 10, 10, 10, 9, NA, ...
$ review_scores_location      <int> NA, NA, 10, 10, NA, 9, 10, NA, NA, NA, 10, 10, 9, 10, NA, ...
$ review_scores_value         <int> NA, NA, 10, 10, NA, 9, 10, NA, NA, NA, 10, 10, 8, 9, NA, 1...
$ cancellation_policy         <chr> "moderate", "flexible", "flexible", "moderate", "flexible"...
$ listing_url                 <chr> "https://www.airbnb.com/rooms/18292044", "https://www.airb...
$ description                 <chr> "NBA player and China star Huge residence apartment, Hugel...

What you see within the angular brackets <...> next to each column name is the data type of that variable.

But what are data types? And, why are they important? What exactly is a variable in programming? What is a function?

Let’s take a detour.

Detour: variables, data types, and functions in R

Variables (programming)

In programming, a variable stores some value (not to be confused with what we call variable in statistics). Our code can then reference the name of the variable for different purposes.

We use the symbol <- (less than sign followed by hyphen) to assign value to a variable name. For example, in the code below, foo and bar are variable names. We assigned the values 43 and "kitten" to foo and bar respectively. You can read them as foo gets 43 and bar gets "kitten".

foo <- 43
bar <- "kitten"
print(foo * 10)

[1] 430

Important: I’ve been using the word variable to mean two different things. A variable in programming is what I described above. A variable in statistics is an attribute of something, whatever the data is about. For example, if the data is about people, some variables could be age, sex, income, address. The names of the columns of a dataframe are variables in the statistical sense. Usually, the context would clarify which meaning is intended.

Tip: Give short descriptive names to your variables. This will make the code easy to follow.

Exercise: a room is 11 yards long and 7.5 yards wide. Assign these values to width and length variables and then calculate the area of the room. You can, of course, give different names to these variables if you wish.

Data types

To understand how R stores and handles information, we need to know a little about data types.

But, first, vector: one of the key concepts in R. Think of a vector as simply a sequence of data. For example, a column in a data table is a vector. This is how you create a vector: c(element1, element2, element3, ...)

vec <- c(3, 5, 10, 20)
print(vec)

[1]  3  5 10 20

A vector can contain four main types of data:

• Integer: such as 2, 543, 90.
• Double: numbers other than integers, such as 4.56, 1/33. Doubles are always approximations.
• Character: a sequence of letters, numbers, punctuations, etc., such as “rainbow”, “34265”. Character data are always written within quotes, either single or double (we’ll always use double for consistency).
• Logical: data that can take on one of only two values — TRUE or FALSE.

Discuss: Why is “34265” of character type? What is a real world example of this kind of data?

There are two other important ways to store data: - factor, for categorical variables (here “variable” in statistical sense). A categorical variable is one that can take on a limited fixed number of values. - date-time or just date

Functions

The concept of a function is exactly the same as in Excel. You give some values (called parameters) to a function. The function does something with the values and returns a new value.

Here’s a simple function that we named add10. All it does is adds 10 to any number given to it.

add10 <-  function(num){
  new_value <-  num + 10
  return (new_value)
}

And, then, we can use this function whenever we need to add 10 to a number (Of course, we won’t. There are easier ways to add a number).

# Add 10 to 35
add10(35)

[1] 45

Today, we won’t write any function, Rather, we’ll use functions written by others. Here’s an example: the mean function, which comes preloaded in R.

# First let's create a vector
vec2 <- c(23, 53, 11, 34, 87, 100, 5, 12, 66, 9, 87, 110, 20, 33, 54, 43, 76)
# Let's calculate the mean of the vector
calculated_mean <- mean(vec2)
# Then output it with some text description
message("The mean of the vector vec2 is ", mean(calculated_mean))

The mean of the vector vec2 is 48.4117647058824

*Exercise: calculate the sum, median, and standard deviation (hint: sd) of the vector vec2.

Now that we understand data types, let’s load some new data.

Exercise: You’ll see there’s another data file in the folder: “us_babynames.csv”. Load the data from this file to a dataframe (hint: read_csv). Assign the dataframe to a variable named df_babynames. We’ll come back to this dataframe later.

Exercise: Now take a look at the dataframe’s data types. (hint: glimpse)

Back to analysis

Fix data types

Let’s take another look at the data types of our dataframe.

glimpse(df)

Observations: 40,752
Variables: 28
$ host_id                     <int> 58306608, 124280354, 124280354, 124280354, 124280354, 1242...
$ host_since                  <chr> "2/11/2016", "4/4/2017", "4/4/2017", "4/4/2017", "4/4/2017...
$ host_response_time          <chr> "within an hour", "within an hour", "within an hour", "wit...
$ host_response_rate          <chr> "0.78", "0.99", "0.99", "0.99", "0.99", "0.99", "0.95", "N...
$ host_listings_count         <int> 1, 7, 7, 7, 7, 7, 1, 1, 1, 7, 1, 1, 1, 1, 1, 1, 2, 2, 1, 1...
$ neighbourhood               <chr> "Hell's Kitchen", "Hell's Kitchen", "Hell's Kitchen", "Hel...
$ borough                     <chr> "Manhattan", "Manhattan", "Manhattan", "Manhattan", "Manha...
$ zip_code                    <int> 10000, 10001, 10001, 10001, 10001, 10001, 10001, 10001, 10...
$ property_type               <chr> "Apartment", "Apartment", "Apartment", "Apartment", "Apart...
$ room_type                   <chr> "Private room", "Shared room", "Shared room", "Shared room...
$ accommodates                <int> 2, 1, 1, 1, 1, 1, 2, 1, 1, 1, 2, 2, 1, 2, 1, 1, 2, 2, 1, 1...
$ bathrooms                   <dbl> 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 0.5, 3.0, 1.0, 1.0, 1.0, 1.0...
$ bedrooms                    <int> 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1...
$ beds                        <int> 1, 1, 1, 1, 1, 1, 2, 6, 1, 1, 1, 1, 1, 1, 1, 1, 2, 1, 1, 1...
$ amenities                   <chr> "{TV,Internet,\"Wireless Internet\",\"Air conditioning\",P...
$ price                       <dbl> 200, 28, 39, 39, 39, 43, 50, 50, 50, 53, 54, 55, 55, 60, 6...
$ calendar_updated            <chr> "never", "5 days ago", "5 days ago", "3 days ago", "6 days...
$ number_of_reviews           <int> 0, 0, 2, 1, 0, 3, 185, 0, 0, 0, 21, 5, 3, 12, 0, 20, 1, 1,...
$ review_scores_rating        <int> NA, NA, 100, 100, NA, 87, 93, NA, NA, NA, 96, 100, 80, 92,...
$ review_scores_accuracy      <int> NA, NA, 10, 10, NA, 9, 9, NA, NA, NA, 10, 10, 10, 9, NA, 1...
$ review_scores_cleanliness   <int> NA, NA, 10, 10, NA, 8, 9, NA, NA, NA, 10, 8, 8, 9, NA, 9, ...
$ review_scores_checkin       <int> NA, NA, 10, 10, NA, 9, 10, NA, NA, NA, 10, 10, 10, 9, NA, ...
$ review_scores_communication <int> NA, NA, 10, 10, NA, 9, 10, NA, NA, NA, 10, 10, 10, 9, NA, ...
$ review_scores_location      <int> NA, NA, 10, 10, NA, 9, 10, NA, NA, NA, 10, 10, 9, 10, NA, ...
$ review_scores_value         <int> NA, NA, 10, 10, NA, 9, 10, NA, NA, NA, 10, 10, 8, 9, NA, 1...
$ cancellation_policy         <chr> "moderate", "flexible", "flexible", "moderate", "flexible"...
$ listing_url                 <chr> "https://www.airbnb.com/rooms/18292044", "https://www.airb...
$ description                 <chr> "NBA player and China star Huge residence apartment, Hugel...

Some of the data types don’t look right. Let’s correct them.

Discuss: why is it important to have the correct data type?

But, first, let’s learn how to reference a column in a dataframe: the name of the dataframe followed by $ and then the column name. For example: df$cancellation_policy or df_babynames$Gender.

To change data type, we apply the relevant function, such as.character or as.factor, to a column and then assign the output of the function (values with changed data type) to the same column.

df$host_id <- as.character(df$host_id)
df$host_response_rate <- as.double(df$host_response_rate)

NAs introduced by coercion

df$property_type <- as.factor(df$property_type)
df$host_since <- mdy(df$host_since)

Side note: The last function mdy is different from others. Normally, working with date-time in R (and in prgramming in general) is not an easy task. The lubridate library, which we loaded earlier, makes the job much easier. We’ll not go into the details of date-times. What we did here with mdy is convert character data into date data. We used mdy because the character data were in the form month-date-year. If they were in the form ,for example, year-month-date, we would’ve used the function ymd.

Exercise: Change the data type of any other variables you think necessary.

Missing values

Some values will almost inevitably be missing in a medium to large dataset. Let’s inspect our data for missing values using the function summary, which will give us plenty of other important information.

summary(df)

   host_id            host_since         host_response_time host_response_rate host_listings_count
 Length:40752       Min.   :2008-03-04   Length:40752       Min.   :0.030      Min.   :  0.000    
 Class :character   1st Qu.:2013-01-28   Class :character   1st Qu.:0.940      1st Qu.:  1.000    
 Mode  :character   Median :2014-07-18   Mode  :character   Median :1.000      Median :  1.000    
                    Mean   :2014-05-11                      Mean   :0.934      Mean   :  2.221    
                    3rd Qu.:2015-10-20                      3rd Qu.:1.000      3rd Qu.:  2.000    
                    Max.   :2017-05-02                      Max.   :1.000      Max.   :855.000    
                    NA's   :251                             NA's   :13791      NA's   :251        
 neighbourhood        borough             zip_code         property_type    room_type        
 Length:40752       Length:40752       Min.   :10000   Apartment  :34980   Length:40752      
 Class :character   Class :character   1st Qu.:10023   House      : 3393   Class :character  
 Mode  :character   Mode  :character   Median :11101   Loft       :  875   Mode  :character  
                                       Mean   :10643   Townhouse  :  562                     
                                       3rd Qu.:11220   Condominium:  357                     
                                       Max.   :11694   Other      :  209                     
                                       NA's   :614     (Other)    :  376                     
  accommodates      bathrooms        bedrooms           beds         amenities        
 Min.   : 1.000   Min.   :0.000   Min.   : 0.000   Min.   : 1.000   Length:40752      
 1st Qu.: 2.000   1st Qu.:1.000   1st Qu.: 1.000   1st Qu.: 1.000   Class :character  
 Median : 2.000   Median :1.000   Median : 1.000   Median : 1.000   Mode  :character  
 Mean   : 2.789   Mean   :1.126   Mean   : 1.149   Mean   : 1.534                     
 3rd Qu.: 4.000   3rd Qu.:1.000   3rd Qu.: 1.000   3rd Qu.: 2.000                     
 Max.   :16.000   Max.   :8.000   Max.   :10.000   Max.   :16.000                     
                  NA's   :159     NA's   :69       NA's   :79                         
     price         calendar_updated   number_of_reviews review_scores_rating
 Min.   :   10.0   Length:40752       Min.   :  0.00    Min.   : 20         
 1st Qu.:   70.0   Class :character   1st Qu.:  1.00    1st Qu.: 90         
 Median :  100.0   Mode  :character   Median :  4.00    Median : 95         
 Mean   :  145.3                      Mean   : 16.41    Mean   : 93         
 3rd Qu.:  170.0                      3rd Qu.: 18.00    3rd Qu.:100         
 Max.   :10000.0                      Max.   :432.00    Max.   :100         
                                                        NA's   :9924        
 review_scores_accuracy review_scores_cleanliness review_scores_checkin
 Min.   : 2.000         Min.   : 2.000            Min.   : 2.000       
 1st Qu.: 9.000         1st Qu.: 9.000            1st Qu.:10.000       
 Median :10.000         Median :10.000            Median :10.000       
 Mean   : 9.532         Mean   : 9.196            Mean   : 9.709       
 3rd Qu.:10.000         3rd Qu.:10.000            3rd Qu.:10.000       
 Max.   :10.000         Max.   :10.000            Max.   :10.000       
 NA's   :10017          NA's   :9985              NA's   :10036        
 review_scores_communication review_scores_location review_scores_value cancellation_policy
 Min.   : 2.000              Min.   : 2.000         Min.   : 2.000      Length:40752       
 1st Qu.:10.000              1st Qu.: 9.000         1st Qu.: 9.000      Class :character   
 Median :10.000              Median :10.000         Median :10.000      Mode  :character   
 Mean   : 9.738              Mean   : 9.418         Mean   : 9.338                         
 3rd Qu.:10.000              3rd Qu.:10.000         3rd Qu.:10.000                         
 Max.   :10.000              Max.   :10.000         Max.   :10.000                         
 NA's   :9986                NA's   :10037          NA's   :10039                          
 listing_url        description       
 Length:40752       Length:40752      
 Class :character   Class :character  
 Mode  :character   Mode  :character  
                                      
                                      
                                      
                                      

There are a few options for dealing with missing values.

• We can drop all rows that have one or more missing values.
• Drop rows that have missing values in particular columns.
• Replace the missing values with some other value.
• Do nothing, but do remember to take care of them when running arithmetic operations. (explained later)

In our case, we’ll go with the last option.

Discuss: Is the last option the best for our dataset? What would be a better alternative?

Data manipluation and exploration

Finally, we’re ready to dive into the data. We’ll use six functions — think of them as six verbs — to slice and dice the data in all kinds of ways. These functions come from the dplyr library. They are:

• filter
• select
• mutate
• arrange
• summarize
• group_by

We’ll first learn these functions one by one and later we’ll learn how to combine them for powerful analysis.

For each of these functions, we provide the name of the dataframe as the first parameter. The subsequent parameters inform what the function is to do with the dataframe.

filter

You use filter, when you want a subset of the rows based on one or more conditions. The returned rows will be those that meet the condition(s).

The syntax is: filter(name_of_dataframe, condition1, condition2, ...)

Use these logical operators to create the conditions:

< less than <= less than or equal to > greater than >= greater than or equal to == exactly equal to != not equal to !x Not x x | y x OR y x & y x AND y

Example: Return a dataframe with only those rows where the neighbourhood is Chelsea.

filter(df, neighbourhood == "Chelsea")

Example: Return a dataframe with only those rows where the price is more than 5000.

You can combine multiple conditions.

Example: Return a dataframe with rows where borough is Manhattan and price is less than 50.

Example: Return a dataframe with rows where cancellation_policy is flexible or accommodates more than 2.

Example: Return a dataframe with rows where number_of_reviews is not 0

filter(df, number_of_reviews != 0)

Note: when multiple conditions are separated by commas are assumed to have and logical operator. Using & instead would be the same thing. For or and other logical operators, we have to explicitly use the appropriate logical operator (e.g. | for or).

Exercise: Return a dataframe with rows where number of bedrooms is not 1 and property_type is House

Exercise: Return a dataframe with rows where host_response_time is within an hour or host_response_rate is more than 90% or calendar was updated today

select

Our second verb is select, which is used to return a subset of the columns.

The syntax is: select(name_of_dataframe, name_of_column1, name_of_column2, ...)

There are some variations to this syntax, which come in handy in certain situations.

• If you want all columns except one: select(name_of_dataframe, -column_to_exclude)
• If you want all columns except two: select(name_of_dataframe, -c(column_to_exlude1, column_to_exclue2))
• If you want the third through eigth columns: select(name_of_dataframe, 3:8)

Example: Return a dataframe with only the listing_url column

Example: Return a dataframe with columns host_response_time, cancellation_policy, and neighbourhood

Example: Return a dataframe with second through fourth columns

Exercise: Return a dataframe with property_type, room_type, and zip_code columns.

Exercise: Return a dataframe with the last 23 columns (sixth through 28th)

Exercise: What would be another way of getting the same dataframe?

Detour: piping

Before I introduce the third verb, let’s take another detour.

You can’t accomplish a lot with filter and select in isolation. Let’s combine them.

Here, we’re creating a new dataframe df_expensive by using filter and then we’re selecting some columns of interest from that dataframe.

Discuss: More than $8000 per night? What’s going on here? How can we find out more? What should we do we this kind of outliers?

The above code could be written more succinctly by using pipe %>%.

df %>% 
  filter(price > 8000) %>% 
  select(price, neighbourhood, description, listing_url)

Let’s try to understand how the pipe works.

filter and select both receive a dataframe as their first parameter (this is true for the other four main verbs as well.) Also, you may have noticed that filter and select return a dataframe (again, it’s true for the others as well). Thanks to this, we can use the pipe to string together several verbs to form a complex query.

filter(df, price > 8000)

is the same as

df %>% filter(price > 8000)

Also,

select(df, price, review_scores_rating, listing_url)

is the same as

df %>% select(price, review_scores_rating, listing_url)

The left side of the pipe is used as the first parameter of the function on the right side of the pipe.

This is how filter, select, and such functions receive a dataframe, does something to it, returns the modified dataframe and passes it on to the next function.

Here’s again the piped expression. Make sure you understand what’s going on.

df %>% 
  filter(price > 8000) %>% 
  select(price, review_scores_rating, listing_url)

arrange

arrange is our third verb. (Note that I’m using verb and function interchangeably.) arrange sorts a dataframe based on the values of one or more columns.

Example: Get the same dataframe as the last code chunk. Sort it by price.

df %>% 
  filter(price > 8000) %>% 
  select(price, neighbourhood, listing_url) %>% 
  arrange(price)

Example: The same dataframe, but now in descending order of price.

df %>% 
  filter(price > 8000) %>% 
  select(price, neighbourhood, listing_url) %>% 
  arrange(desc(price))

Example: You can add more columns to arrange. Now when price is the same, the order will be based on neighbourhood.

df %>% 
  filter(price > 8000) %>% 
  select(price, neighbourhood, listing_url) %>% 
  arrange(desc(price), desc(neighbourhood))

Exercise: Return the whole dataframe df, in descending order of number of bedrooms and where number of bedrooms are equal, descending order of review_scores_rating

mutate

mutate creates a new column by modifying one or more existing columns. It’s better explained by examples.

Example:

Exercise: Create a new review_rating column, which is an average of review_scores_accuracy and review_scores_value.

summarize (or summarise)

summarize, as you’d expect, gives some kind of summary (such as mean, median, min, max) of the values of a given column.

Example: What is the total number of people that can stay in NYC Airbnbs?

df %>% 
  summarize(total_accommodates = sum(accommodates))

Here, total_accommodates is a name I’ve given to the summary number.

The common summary functions are: - sum - n (count) - mean - median - sd (standard deviation) - var (variance) - range - min - max

Note: The function n doesn’t need a parameter, because it just counts the number of rows in the dataframe. This would become more clear in the examples.

We can also write our own function and use here. But that is a more advanced topic.

Example: What is the mean rating for accuracy?

summarize(df, mean_accuracy = mean(review_scores_accuracy))

This did not give us what we expected. This is because any arithmetic operation involving NA (that, not available or missing values) results in NA.

5 + NA

Or,

NA / 33

The review_scores_accuracy had several NAs. The correct way of doing this would be:

summarize(df, mean_accuracy = mean(review_scores_accuracy, na.rm = TRUE))

na.rm = TRUE means “remove all NAs before performing arithmetic operations.”

When we summed the accommodates column earlier, we didn’t add na.rm = TRUE because that column didn’t have any NAs.

Example: We can have multiple summaries at the same time.

summarize(df, min_score_value = min(review_scores_value, na.rm = TRUE), 
          max_score_value = max(review_scores_value, na.rm = TRUE))

Example: What’s the total number of listings in this dataset?

Exercise: The host who’s been with Airbnb NYC for the longest time started on which date?

Exercise: Find the standard deviation and mean of the price variable.

Exercise: Count the number of unique zip_codes in the data. Hint distinct and n.

Exercise: How many listings have perfect 100 for review_scores_rating (which is the overall rating shown as stars on the listing’s webpage)?

Exercise: How many listings have perfect 100 for review_scores_rating and price less than 50?

group-by

group-by is one of the most useful functions. It divides the data into different groups and then summarize summarizes each of the group. This way we can compre the different groups on various attributes.

Example: What’s the mean and median price in each of the five boroughs?

df %>% 
  group_by(borough) %>% 
  summarize(mean(price), median(price))

Exercise: Return the same dataframe but sorted by mean price.

Example: Which are the top 10 neighbourhoods by total number of listings?

df %>% 
  group_by(neighbourhood) %>% 
  summarize(count = n()) %>% 
  arrange(desc(count)) %>% 
  top_n(10)

We introduce a new function top_n here, which is self-explanatory.

Exercise: Do the same as above but only for listings where price is more than 500.

Example: In Manhattan, which neighbourhood has the highest mean price?*

Exercise: In Bronx and Queens, what’s the mean price for different property_type?

*Exercise: Which combination of property_type and room_type has the highest median price?

Here are some excellent resources if you want to keep learning R:

• http://tryr.codeschool.com
• http://r4ds.had.co.nz/
• http://swirlstats.com/